Skip to content

HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions#6885

Merged
steveloughran merged 2 commits intoapache:trunkfrom
steveloughran:s3/HADOOP-19203-bulk-delete-exceptions
Jun 19, 2024
Merged

HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions#6885
steveloughran merged 2 commits intoapache:trunkfrom
steveloughran:s3/HADOOP-19203-bulk-delete-exceptions

Conversation

@steveloughran
Copy link
Contributor

@steveloughran steveloughran commented Jun 12, 2024

-WrappedIO methods raise UncheckedIOEs
-new class org.apache.hadoop.util.functional.FunctionalIO
with wrap/unwrap and the ability to generate a
java.util.function.Supplier around a CallableRaisingIOE.
-Tests

How was this patch tested?

  • New test for the wrapping logic
  • bulk delete contract tests against s3 and abfs

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@steveloughran
HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOEx…
9581c1b 
…ceptions

-WrappedIO methods raise UncheckIOEs
-new class org.apache.hadoop.util.functional.FunctionalIO
 with wrap/unwrap and the ability to generate a
 java.util.function.Supplier around a CallableRaisingIOE.
-Tests

Change-Id: Icad3bfa30bd5226a5fb6534227e9e56e4b37d536
@hadoop-yetus
Copy link

hadoop-yetus commented Jun 12, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 32m 43s trunk passed
+1 💚 compile 8m 55s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 compile 8m 6s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 checkstyle 0m 41s trunk passed
+1 💚 mvnsite 0m 57s trunk passed
+1 💚 javadoc 0m 41s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 0m 32s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 1m 28s trunk passed
+1 💚 shadedclient 21m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 31s the patch passed
+1 💚 compile 8m 24s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javac 8m 24s the patch passed
+1 💚 compile 8m 12s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 javac 8m 12s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 38s /results-checkstyle-hadoop-common-project_hadoop-common.txt hadoop-common-project/hadoop-common: The patch generated 3 new + 3 unchanged - 1 fixed = 6 total (was 4)
+1 💚 mvnsite 0m 53s the patch passed
+1 💚 javadoc 0m 40s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 0m 35s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 1m 33s the patch passed
+1 💚 shadedclient 21m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 32s hadoop-common in the patch passed.
+1 💚 asflicense 0m 39s The patch does not generate ASF License warnings.
136m 27s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/1/artifact/out/Dockerfile
GITHUB PR #6885
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux e38782558de5 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 9581c1b
Default Java Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/1/testReport/
Max. process+thread count 1279 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
HADOOP-19203. Style checking
e9c38f5 
Change-Id: Ica826512b0efc4da0edd45de031116508198a5b2
@hadoop-yetus
Copy link

hadoop-yetus commented Jun 13, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 38m 11s trunk passed
+1 💚 compile 10m 32s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 compile 9m 22s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 checkstyle 0m 48s trunk passed
+1 💚 mvnsite 1m 2s trunk passed
+1 💚 javadoc 0m 48s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 0m 34s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 1m 34s trunk passed
+1 💚 shadedclient 24m 51s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 33s the patch passed
+1 💚 compile 9m 42s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javac 9m 42s the patch passed
+1 💚 compile 8m 59s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 javac 8m 59s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 39s /results-checkstyle-hadoop-common-project_hadoop-common.txt hadoop-common-project/hadoop-common: The patch generated 1 new + 3 unchanged - 1 fixed = 4 total (was 4)
+1 💚 mvnsite 0m 58s the patch passed
+1 💚 javadoc 0m 45s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 0m 33s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 1m 39s the patch passed
+1 💚 shadedclient 25m 7s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 29s hadoop-common in the patch passed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
154m 55s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/2/artifact/out/Dockerfile
GITHUB PR #6885
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux e3e2e3284b42 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e9c38f5
Default Java Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/2/testReport/
Max. process+thread count 1264 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor Author

steveloughran commented Jun 14, 2024

checkstyle is complaining about the _ in the method names.

@mukund-thakur this should be good for review.

@steveloughran steveloughran mentioned this pull request Jun 14, 2024
Merged
4 tasks
mukund-thakur
mukund-thakur approved these changes Jun 17, 2024
View reviewed changes
Copy link
Contributor

@mukund-thakur mukund-thakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

...mmon-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FunctionalIO.java
* propagated unchanged.
* @param <T> type of result
*/
private static final class UncheckedIOExceptionSupplier<T> implements Supplier<T> {
Copy link
Contributor

@mukund-thakur mukund-thakur Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any use of this so for though. Assuming it will be used later?

Copy link
Contributor Author

@steveloughran steveloughran Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest of wrapped IO. and actually just done something identical in #6892

@steveloughran steveloughran merged commit 8ac9c18 into apache:trunk Jun 19, 2024
steveloughran added a commit that referenced this pull request Jun 20, 2024
@steveloughran
HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOEx…
b8a3901 
…ceptions (#6885)

* WrappedIO methods raise UncheckedIOExceptions
*New class org.apache.hadoop.util.functional.FunctionalIO
 with wrap/unwrap and the ability to generate a
 java.util.function.Supplier around a CallableRaisingIOE.

Contributed by Steve Loughran
KeeProMise pushed a commit to KeeProMise/hadoop that referenced this pull request Sep 9, 2024
@steveloughran @KeeProMise
HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOEx…
26393b7 
…ceptions (apache#6885)



* WrappedIO methods raise UncheckedIOExceptions
*New class org.apache.hadoop.util.functional.FunctionalIO
 with wrap/unwrap and the ability to generate a
 java.util.function.Supplier around a CallableRaisingIOE.

Contributed by Steve Loughran
Hexiaoqiao pushed a commit to Hexiaoqiao/hadoop that referenced this pull request Sep 12, 2024
@steveloughran @Hexiaoqiao
HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOEx…
6ba39a9 
…ceptions (apache#6885)



* WrappedIO methods raise UncheckedIOExceptions
*New class org.apache.hadoop.util.functional.FunctionalIO
 with wrap/unwrap and the ability to generate a
 java.util.function.Supplier around a CallableRaisingIOE.

Contributed by Steve Loughran
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mukund-thakur mukund-thakur mukund-thakur approved these changes

Assignees

No one assigned

Labels

Common trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@steveloughran @hadoop-yetus @mukund-thakur